These pictures are usually produced using a so-called "interlaced scan" standard; each picture frame is split into two successive fields, which are offset in space by one half line in the vertical direction and which are offset in time by the time required to scan one field.
Interlacing suffers from several drawbacks. It considerably complicates: converting pictures from one television standard to another; space-time analysis of pictures; and also improving pictures by applying image processing techniques thereto. Image information has a three dimensional structure (two space dimensions, plus time) and when scanning is interlaced this structure is not orthogonal; a given point in a field has no equivalent in the following field; with the following field only containing points that are adjacent to the given point, with said adjacent points being absent from the field containing the given point.
It is therefore necessary to perform "deinterlacing", i.e. to convert from interlaced scanning using two successive fields to sequential scanning in which successive frames are identical in structure and superposable.
Deinterlacing makes use of interpolation between image points in space-time. Each image point or "pixel" is defined by two space co-ordinates (x along horizontal lines and y in a vertical direction perpendicular to the lines) together with a time co-ordinate (t) associated with the interlaced field or non-interlaced frame to which the pixel belongs.
There are numerous studies concerning deinterlacing, and the following may be mentioned:
J. Santillana, B. Choquet, "Algorithmes de detection et d'estimation de mouvement dans des images de television" (Algorithms for detecting and estimating motion in television pictures), CCETT: RTI/T/007/85/JS-BC, March 85.
B. Choquet, "Inventaire des techniques de traitement d'images en conversion de normes de representation pour la television Haute Definition" (Listt of image processing techniques for conversion between picture standards in HDTV), CCETT: RTI/NT/031/86/BC, Nov. 25, 1986.
D. Pele, B. Choquet, "Algorithmes d'analyse spatio-temporelle pour Television Haute Definition (Algorithms for space-time analysis in HDTV), CCETT: CAL/T/002/87/DP and RTI/T/004/87/BC, April 87.
B. Choquet, P. Siohan, A. Benslimane, "Detection d'activite, dans une sequence d'images, par gradient temporel et par gradient spatial" (Detecting activity in a sequence of images by means of a time gradient and a space gradient), CCETT: RTI/T/017/85 BC and CAL/T/006/85/PS.AB, July 85.
P. Siohan, B. Choquet, "Multidimensional processing for motion detection," PCS 86: Tokyo, April 2-4, pages 65, 66, session 3.18.
B. Choquet, P. Siohan, "Enhancement techniques of a motion detector in High Definition Television," IEE, Second International Conference on Image processing and its applications, Conference publication number 265, June 24-26, 1986 (pages 215-219).
P. Siohan, B. Choquet, "Motion detection for the deinterlace in HDTV picture processing," International Eurasip Workshop on Coding of HDTV: Nov. 12-13, 1986, l'Aquila, Italy (Vol. 2).
B. Choquet, P. Siohan, "Desentrelacement par interpolations adaptatives" (Deinterlacing by adaptive interpolation), Colloquim on HDTV, Ottawa, Oct. 4-8, 1986.
D. Pele, "Quelques methodes de segmentation d'images" (Some image segmentation methods), ENST-Bretagne: 86/LEMPAB/RCM/01.
In spite of these numerous studies, there is no deinterlace technique available at present which is simultaneously simple, fast, and effective, in particular for high definition television (HDTV) for which the image quality required in base band means that the techniques used in the past need to be thoroughly overhauled.
The criterion of simplicity relates to the number of data items that need to be taken into account when performing deinterlacing. Proposals have been made, for example, to use a lowpass linear filter whose input information is constituted by 11 frame lines (vertical aperture) over 7 fields (time aperture). It is clear that such a lowpass linear filter is at the limit of acceptable complexity.
The speed criterion relates to the speed at which the means used can act: naturally it is necessary for them to be capable of operating in real time, i.e. at least as fast as the rate at which picture information is renewed. This leads to a compromise between the amount of input information which can be taken into account and the sophistication of the means which process said information. For example, in the above-mentioned linear filter, it is practically impossible to change the interpolation parameters as a function of the local contents of a picture.
The effectiveness criterion relates to the results obtained on deinterlaced pictures. In particular, the following defects must be avoided:
loss of definition in non-moving zones of the picture, which phenomenon is particularly critical and unacceptable for an observer; PA1 loss of definition in zones which are in motion, which becomes unacceptable whenever the eye is following the moving object; PA1 diagonal outlines which are degraded since they are reproduced in "staircase" form; and PA1 poor reproduction or total loss of outlines which are moving quickly or very quickly. PA1 (a) selecting a time pair and a space pair of main points about a desired point in a line to be interpolated in a field being interpolated, the main points of said time pair having the same position in the image as the desired point, but being situated in the two fields occurring immediately before and immediately after the field being interpolated, while the main points of the space pair come from the field being interpolated, from respective ones of the two positions which are vertically adjacent to the position of the desired point; PA1 (b) calculating the "time" main difference between the signals at the two points in the time pair, and also the "space" main difference between the signals at the two points in the space pair; PA1 (c) selecting an interpolation function taking account of said main differences; and PA1 (d) calculating the interpolated signal of the desired point by means of said interpolation function. PA1 (c1) selecting auxiliary points situated on the same frame line as the main points and suitable for defining auxiliary pairs of points on either side of the desired point, while simultaneously selecting a space-time zone surrounding the desired point and including the two main points and the auxiliary points associated with said two main points; PA1 (c2) calculating auxiliary differences between the signals at the two points of each auxiliary pair; PA1 (c3) determining first level partial interpolation in that one of the directions between the main pair and the associated auxiliary pairs which gives the minimum difference; and PA1 (c4) determining a second level partial interpolation in the direction of the pair of main points concerned. PA1 (d1) for each pair of main points, hierarchical selection between first level partial interpolation and second level partial interpolation, as a function of the fact that said zone does not contain or does contain a local outline which is horizontal; and PA1 (d2) calculating the final interpolated luminance of the desired product from the partial interpolations each relating to pairs of main points. PA1 (c4) determining a third level partial interpolation in the direction of that pair of furthest apart auxiliary points which gives a minimum difference. PA1 stage (a) comprises, in addition to the space pair and the time pair, selecting two "space-time" pairs each comprising two additional points on either side of the desired point, each of said additional main points being situated in the same field and in the same vertical position as one of the first-mentioned time main points, but on a frame line which is adjacent to that of said time main point in the field concerned; and PA1 stage (b) includes calculating two space-time main differences with these two pairs of additional main points. PA1 (e) combining each initial field with the corresponding field of interpolated points, thereby obtaining a deinterlaced frame. PA1 an input for sequential picture signals, providing on each occasion a signal relating to a new point situated on a given line of the incident field; PA1 memory means for storing successive input signals in fields and in lines, in order to define a space-time window around a desired point on a line to be interpolated in a field being interpolated and preceding the incident field, said memories simultaneously providing output signals representing a time pair of main points having the same position as the desired point but respectively occupying the field immediately before and the field immediately after the field being interpolated, and a space pair of main points which are in the field being interpolated but in respective ones of the two positions vertically adjacent to the desired point; PA1 first processing means for establishing temporary interpolations of the desired point on the basis of the outputs from the memories (the word "temporary" is used here since the prior art does not include partial interpolation in the meaning of the present invention); and PA1 second processing means also operating on the basis on the outputs from the memories in order to establish the final interpolation of the desired point from the temporary interpolations, by means of differences between the main points of said time and space pairs. PA1 determine auxiliary differences between the pairs of auxiliary points associated with said main points and on either side of the desired point in space time; PA1 calculating a first level partial interpolation in the direction of that one of the main pair and the associated auxiliary pairs which gives a minimum difference; and PA1 calculating a second level partial interpolation in the direction of the pair of main points concerned; and PA1 preferably, also calculating a third level partial interpolation in the direction of the furthest-apart pair of auxiliary points giving rise to a minimum difference; PA1 with the second processing means providing a partial interpolation for each pair of main points by performing hierarchical selection between first level interpolation and second level interpolation (and optionally third level interpolation) as a function of a search for a minimum amongst the differences or amongst normed linear combinations of the differences relating to said pair of main points and to the associated auxiliary points.
The object of the present invention is to provide an improved method and apparatus for processing image signals, and suitable, in particular, for substantially improving the present situation in deinterlacing techniques.
The present invention is based on a method comprising the following stages:
In other words, it is known to take video input signals for the purpose firstly of detecting motion in a zone surrounding a point to be interpolated and secondly for selecting an interpolation function from a library of such functions. These two types of information are used for deciding on the choice of a given aglorithm and for establishing transition criteria between two different algorithms. This gives rise to a final interpolation function which is applied to the input signals or to signals derived therefrom.
For example, motion may be detected on the basis of the time main difference and the space main difference, for which various proposals have already been made in the prior art.